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EXECUTIVE  SUMMARY 


Requirement : 

The  solicitation  required  that  Florida  State  University  (FSU)  Develop  a 
set  of  evaluation  standards  that  will  determine  whether  the  system: 

(1)  meets  the  stated  specifications  for  courseware,  software  and 
hardware; 

(2)  is  as  effective  as  the  functional  BSEP  II  program;  and 

(3)  is  effective  in  improving  performance  on  the  job. 


Procedure : 


After  a  literature  review,  an  initial  draft  of  JSEP  goals,  process  steps 
a'nd  performance  standards  objectives  *ss  formulated  by  the  J3EF  interna', 
evaluation  team.  The  final  araft  was  attained  through  an  iterative  revie*, 
process  involving  CET/Hazeltine  designers  and  evaluators  along  with  project 
officers  from  the  Army  Research  Institute. 


Findings: 

While  there  are  several  models  for  expressing  evaluation  standards 
available  in  the  technical  literature,  the  form  of  the  standards  expressed 
herein  is  optimal  for  the  purpose  of  the  project,  namely  the  development  of  a 
job  related,  computer-based  curriculum.  The  objectives  and  process  steps  are 
consistent  with  the  procedures  required  to  design  and  develop  the  JSEP 
curriculum.  The  process  steps  with  their  accompanying  performance  standards 
will  provide  quality  assurance  checkpoints  in  the  development  of  the 
instructional  delivery  system. 


Utilization  of  Findings: 

The  objectives,  process  steps  and  standards  will  be  used  to  cocument  the 
attainment  of  milestones  and  to  demonstrate  critical  steps  in  the  development 
of  computer-based  curricula.  The  objectives  with  standards  will  also  be  points 
of  departure  for  the  conduct  of  Tasks  7,  8,  9,  10,  13,  16,  17,  and  18. 
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FOREWORD 


The  Job  Skills  Education  Program  (JSEP)  is  a  multi-phase  program  begun  in 
Fiscal  Year  1982,  and  designed  to  enhance  enlisted  career  potential  by 
improving  soldier  job  performance.  The  sponsor,  the  Education  Division, 

Office  of  the  Deputy  Chief  of  Staff  for  Personnel,  expects  JSEP  to  replace 
the  Army's  current  Basic  Skills  Education  Program  when  it  is  implemented. 

The  JSEP  program,  being  developed  by  Florida  State  University  (FSU)  will 
result  in  a  standardized  curriculum  for  soldiers  who  demonstrate  deficiencies 
in  the  knowledge  and  skills  required  to  successfully  learn  their  Military 
Occupational  Specialty  (MOS). 

In  accordance  with  current  policy,  JSEP  will  be  an  on-duty  program.  It  will 
also  use  a  computer-based  management  system  to  facilitate  an  open  entry /open 
exit  approach.  At  present,  most  of  the  lessons  being  developed  will  be 
computer  delivered;  however,  the  plan  calls  for  using  existing  materials,  and 
incorporating  materials  developed  as  part  of  other  ARI  efforts,  whenever 
appropriate . 

A  unique  aspect  of  JSEP  is  that  it  builds  upon  a  very  detailed  front-end 
analysis  of  MOS  Baseline  Skills.  The  analysis  covered  tasks  performed  by 
soldiers  in  the  94  highest  density  MOSs,  in  addition  to  Common  Tasks  (the 
skills  that  all  soldiers,  regardless  of  their  MOS,  need  to  know).  Although  the 
Army  has  over  300  MOSs,  the  94  covered  in  the  analysis  represent  about  80%  of 
all  soldiers.  Perhaps  the  most  useful  product  developed  for  the  analysis  was 
a  taxonomy  listing  more  than  200  prerequisite  competencies. (P.C.)  for  these 
MOSs.  The  competencies  were  derived  from  detailed  reviews  of  Soldier  Manuals, 
and  from  extensive  interviews  with  subject-matter  experts  at  Army  schools. 

This  effort  produced  a  series  of  tests  intended  to  diagnose  deficiencies  in 
the  P.C.s.  Modified  versions  of  these  tests  will  be  used  in  JSEP. 

The  JSEP  program  will  include  a  front-end  learning  strategies  module 
designed  to  improve  soldier  skills  in  reading,  studying,  test  taking,  and 
problem  solving.  The  curriculum  will  consist  of  this  strategies-training,  plus 
180  diagnostic  review  lessons,  and  120  skill  development  lessons,  which  are 
being  developed  for  the  PLATO  and  MicroTICCIT  computer  systems.  The  program  is 
being  tried  out  at  two  TRAD0C  sites  and  two  F0RSC0M  sites,  prior  to  an  Army-wide 
phased  implementation. 
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JSEP  EVALUATION  STANDARDS 


OVERVIEW 


Operational  Problem 

It  Is  not  news  that  soldiers  must  be  trained  to  do  their  jobs.  They  must 
be  trained  so  that  each  Army  job  Is  performed  competent1y--regard1ess  of 
differences  in  ability  and  background  in  newly  entering  soldiers.  To  accept 
less  would  cause  many  mission  elements  to  fail. 

Moreover,  many  Army  jobs  are  increasingly  dependent  upon  the  soldier's 
ability  to  use  high  technology  and  the  ability  to  learn  new  technology  as  it 
develops.  Soldiers,  therefore,  need  more  than  training.  They  need  enough 
education  to  be  able  to  learn  subsequent  jobs,  to  become  eligible  for 
promotion,  and  ultimately,  to  provide  leadership  for  tomorrow's  Army. 

The  Job  Skills  Education  Program  (JSEP)  is  designed  to  provide  soldiers 
with  job-related  basic  skills  instruction  that  is  prerequisite  to  learning 
their  skill  level  1  and  2  job  tasks  during  their  first  duty  assignment.  JSEP 
provides  functional  basic  skills  instruction  on  MOS  specific  requirements.  It 
is  based  on  an  extensive  job  analysis  of  94  of  the  Military  Occupational 
Specialties  (MOS)  which  contain  the  largest  proportion  of  soldiers  and  tasks 
contained  in  the  Soldier's  Manual  of  Common  Tasks. 

As  it  is  conceptualized,  the  JSEP  curriculum  recognizes  that  the  vast 
majority  of  soldiers  will  have  been  exposed  to  similar  basic  skills 
instruction  before  entering  the  Army.  Many  entering  soldiers,  however,  will 
not  have  learned  those  basic  skills  well  enough,  or  will  not  remember  what 
they  have  learned.  To  help  soldiers  learn  better  and  remember  more,  JSEP 
incorporates  straightforward  training  in  research-based  learning  strategies 
that  are  directly  aimed  at  Improving  learning  and  retention. 


Research  Objective 

The  solicitation  required  that  Florida  State  University  (FSU)  develop  a 
set  of  evaluation  standards  that  will  determine  whether  the  system: 

(1)  meets  the  stated  specification  for  courseware,  software  and  hardware 

(2)  is  as  effective  as  a  functional  BSEPII  program;  and  whether 

(3)  is  effective  in  improving  performance  on  the  job. 


Scope 

The  project  purpose,  goal,  objectives  and  process  steps  with  their 
respective  performance  standards  are  presented  for  the  entire  JSEP  project. 

The  performance  standards  are  stated  in  the  form  of  either  process  criteria  or 
in  the  form  of  products  which  (1)  contain  analyses  of  data  or  (2)  describe 
important  processes  used  in  the  development  of  the  curriculum. 
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Approach 


The  strategy  for  developing  JSEP  evaluation  standards  was  first  to 
articulate  goals,  objectives,  and  process  steps  to  describe  the  intents  of  the 
project.  The  works  of  Stake,  Stufflebeam,  Scriven,  Worthen  and  Saunders  were 
important  here.  The  objectives  and  process  steps  were  then  reviewed  by  the 
CET/Hazeltine  JSEP  executive  committee  and  revised  accordingly.  Following 
these  procedures,  an  outline  of  performance  standards  was  initially  developed 
and  these  were  again  reviewed  by  members  of  the  CET/Hazeltine  JSEP  executive 
committee. 


The  performance  standards  are,  in  the  main,  stated  in  the  form  of  either 
process  criteria  or  in  the  form  of  products  which  contain  certain  kinds  of 
information  or  analyses.  For  example,  at  this  time  we  prefer  to  state  that  »e 
will  analyze  and  report  the  pass  rate  of  troops  at  a  field  test  site  while 
controlling  for  a  variety  of  possible  intervening  variables,  rather  than  to 
predict  a  certain  outcome.  We  believe  that  process  standards  must  be  used 
during  the  design  and  development  phases  in  order  to  preserve  the  decision 
freedom  until  empirical  data  are  available  from  the  tryouts.  The  setting  of 
desirable  outcome  standards  will  be  possible  later  when  the  relationship 
between  input  variables  and  outcomes  become  better  known.  Only  at  such  a  time 
can  we  set  a  standard  to  address  the  "How-much-development-is-enough"  issue. 


While  there  are  other  approaches  and  styles  for  the  presentation  of 
evaluation  standards  (e.g.,  Standards  for  Evaluation  of  Educational  Programs, 
P'-piects  and  Materials,  New  vo’"kT  M.cGraw  Hill,  19?!'  the  fo-""  used  he^e,  we 
believe,  both  comprehensive  and  useful  for  making  decisions  for  product 
development  as  well  as  for  internal  and  external  audits  of  project  activities 
and  accomplishments. 


On  the  basis  of  the  review  of  these  evaluation  standards  by  AR I ,  and  their 
subsequent  refinement,  the  evaluation  plan  (Task  7)  will  be  developed  to 
specify  the  methods  of  inquiry,  the  measures  to  be  used,  the  population 
samples,  the  proposed  analyses,  etc.  Until  that  final  refinement,  the 
important  question  is.  What  should  be  observed  about  the  program  that  will  be 
the  most  revealing  for  what  kinds  of  audiences? 

For  now,  we  are  directing  our  inquiry  to  those  who  may  wish  to  know  more 
about  producing  effective  job-related,  computer-based  instruction  for  the 
development  of  basic  intellectual  skills.  Therefore  if  there  is  a  single 
overarching  evaluation  standard,  it  is  the  degree  to  which  JSEP,  as 
systematically  produced  curriculum,  can  effectively  remove  prerequisite 
competency  deficiencies. 


RESULTS  OF  INVESTIGATION 


Purpose 

Provide  an  on-duty,  job-related  basic  skills  development  educational 
program  that  will:  (1)  improve  job  performance,  (2)  function  in  the  Army 
outside  of  immediate  work  setting,  and  (3)  enhance  the  potential  for  completing 
off-duty  educational  programs. 
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Goal 

To  design,  develop  and  test  a  job  and  career  supportive  computer-based 
prerequisite  competency  curriculum  for  possible  Army-wide  adoption. 

Objectives 

1.  Establish  project  review,  control,  and  decision-making  procedures. 

2.  Conduct  a  review  of  literature  and  instructional  materials  related  to  JSEP 
purpose. 

3.  Design  a  job-related,  computer-based  prerequisite  competency  curriculum 
(JSEP). 

4.  Design  an  instructional  support  system  to  field  test  the  JSEP  curriculum. 

5.  Design  a  management  information  system  to  direct,  monitor  and  report 
student  progress  in  JSEP. 

6.  Develop  a  job-related,  computer-based  prerequisite  competency  curriculum. 

7.  Field  test  a  job-related,  computer-based  prerequisite  competency 
curriculum. 


Compare  alternative  computer-based  delivery  systems  in  terms 
effectiveness,  efficiency  and  benefit  to  ll.S.  Army  mission. 


9.  Evaluate  the  potential  impact  of  JSEP  on  the  Army  job  skills,  educational 
programs,  and  soldier  career  goals. 


10.  Explore  the  feasibility  of  using  JSEP  for  the  awarding  of  high  school 
credits  and  credentials. 


11.  Prepare  specifications  for  the  dissemination  of  JSEP  Army-wide. 
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OBJECTIVE  I:  ESTABLISH  PROJECT  REVIEW,  CONTROL,  AND  DECISION-MAKING 
PROCEDURES 


Process  Steps 

1.  Establish  a  JSEP 

Advocacy  Team  representing 
constituent  groups  to 
recommend  policy  regarding 
design,  development,  field 
testing,  evaluation,  and 
dissemination  of  JSEP. 


2.  Establ  ish  an  internal 
review  committee  for  the 
design  and  development  of 
the  JSEP  curriculum. 


3.  Establish  an  internal 
evaluation  committee  to 
assist  the  project  in 
clarifying  project  goals, 
objectives,  and  process 
steps  and  to  help  in 
establishing  criteria  for 
their  attainment. 


4.  Establ  ish  a 

communication  linkage  with 
the  JSEP  external  evaluation 
team. 


Performance  Standards 

la.  The  Advocacy  Team  will 
meet  as  guidance  is 
required  and  will  review 
internal  and  external 
evaluation  of  JSEP 

lb.  Constituencies  represented 
are  TAGO,  TRADOC,  ARI,  Base 
Commanders,  ESOs,  external 
evaluators,  JSEP  developers. 

2a.  The  internal  review 

committee  meets  at  least 
weekly. 


2b.  Internal  review  committee 
consists  of  instructional 
designers,  management 
infc-r.jf'd'.  specialists, 
computer  programmers, 
educational  psychologists, 
evaluators. 

3a.  The  members  of  the 

evaluation  committee  meet 
periodically  with  the 
internal  review 
committee  (see  2b). 

3b.  The  evaluation  committee 
meets  as  guidance  is 
required. 

3c.  The  committee  consists  of 
test  developers,  CTEA 
specialists,  designers, 
management  information 
specialists,  evaluation, 
methodologists. 

4a.  Evaluations  of  JSEP 
products  and  processes 
reviewed  by  external 
evaluators. 

4b.  JSEP  developers  receive 
copies  of  reports  pre- 
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pared  by  external 
evaluators  for  ARI. 

Establish  management 
control  procedures  to 
coordinate  JSEP  design, 
development  and  evaluation 
functions 


5c.  Communication  networks  are 
established  among  developers 
evaluators  and  project 
.  officers. 


5a.  The  management  and  control 
functions  are  described. 

5b.  There  is  a  set  of  policies 
and  procedures  for  manage¬ 
ment  and  control.' 


OBJECTIVE  II:  CONDUCT  A  REVIEW  OF  LITERATURE  AND  INSTRUCTIONAL 
MATERIALS  RELATIVE  TO  JSEP  PURPOSE 


Process  Steps 

1.  Review  MOS  baseline  skills 
analysis  conducted  by  RCA, 
and  identify  prerequisite 
competencies  by  MOS. 

2.  Review  extant  print  and 
computer-based  instruction 
materials  (software)  for 
applicability  to  JSEP 
(especially  those  not 
reviewed  by  TRADOC  SSP'S) 


3.  Conduct  a  literature  review 
of  published  research 
related  to  computer-based 
learning. 


4.  Conduct  a  literature  review 
of  published  research 
related  to  adult  learning 
of  prerequisite 
competencies  and  learning 
strategies. 


5.  Compare  content  of  ETS 
locator  and  diagnostic 
test  items  with  content 
domains  of  prerequisite 
competencies. 


Performance  Standards 

la.  There  is  a  catalogue 

showing  the  relationships 
among  PC's,  MOS  and  task 
indicator  statements. 

2a.  A  list  of  sources  are 
developed. 

2b.  Materials  obtained  are 
recorded. 

2c.  An  evaluation  of  materials 
is  conducted  and  the  out¬ 
come  of  the  evaluation 
recorded. 

3a.  List  of  sources  used  to 
identify  literature  is 
developed. 

3b.  A  bibliography  of  literature 
obtained  for  project  use  is 
kept. 

3c.  A  description  of  how  material 
is  incorporated  into  lessons. 

4a.  A  list  of  potential  sources 
is  developed. 

4b.  A  bibliography  of 

literature  obtained  for 
project  use  is  kept. 

4c.  A  description  of  how  material 
is  incorporated  into  lessons. 

5a.  There  is  documentation 
of  diagnostic  test 
items  for  each  PC  that 
have  been  accepted  as 
is,  modified,  eliminated 
or  replaced  by  CET/ 

Hazeltine  curriculum 
developers. 


OBJECTIVE  III:  DESIGN  A  JOB-RELATED,  COMPUTER-BASED  PREREQUISITE 

J  COMPETENCY  CURRICULUM  (JSEP) 

♦ 

Process  Steps 

Performance  Standards 

1. 

| 

Develop  criteria  for 
specifications  for 
each  JSEP  lesson. 

la. 

Procedures  are  described 
for  how  criteria  .were 
developed  and  adopted. 

1 

lb. 

Lesson  specification  forms 
are  developed  and  approved. 

| 

lc.. 

There  is  a  description  of 
how  each  lesson  directly 
teaches  learning 
strategies  or  enhances 
their  development. 

Id. 

Each  lesson  has  a  valid 
content- pretest  and  posttest. 

le. 

The  use  of  extant  materials 
is  specified  with  copyright 
secured  if  necessary. 

If. 

Alternative  learning 
options  are  specified  that 
supplement  or  complement 

CBI  delivery  system. 

lg. 

MOS-specific  content  is 
outlined. 

2. 

Develop  screen  displays  to 
portray  visual  stimuli  on 

PLATO  and  TICCIT  monitor. 

2a. 

PLATO  and  TICCIT  grids 
completed  and  numbered. 

3. 

De/elop  and  pilot  test  five 
prototype  JSEP  lessons 
using  PLATO  and  TICCIT. 

3a. 

The  purposes  for  developing 
the  specific  prototypes  are 
stated. 

3b. 

The  problems  identified  in 
developing  and  implementing 
the  prototypes  are 
documented. 

3c. 

The  responses  from  the 
target  audiences  are 
documented  and  reviewed. 

3d. 
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The  direct  costs  to  develop 

5  prototypes  are  estimated. 

4.  design  test  interfaces  from 
locator  to  summative  posttest 
for  individualized  JSEP 
program. 


5.  Develop  a  decision  map  (e.g. 
flowchart)  to  guide  learners 
through  JSEP  program. 


6.  Develop  individualized 
summative  posttests  with 
potential  for  generating 
multiple  forms. 


7.  Describe  JSEP,  CBI 
curriculum  model. 


8.  Prepare  for  production  of 
JSEP  curriculum  by 
estimating  resources,  time 
and  cost  for  development. 


4a.  Procedures  routing  students 
through  sequences  of  tests 
are  documented. 

4b.  Options  for  reducing  test 
time  with  cost/benefit 
trade-offs  are  documented. 

5a.  The  development  of  a  decision 
chart  is  described. 

5b.  Flow  chart  is  available 
depicting  learner  flow 
through  the  JSEP 
instructional  system. 

5c.  The  pathways  through  JSEP  is 
founded  in  logical , 
theoretical,  or  empirical 
rationale. 

6a.  Procedures  for  generating 
items  to  form  a  summative 
posttest  are  documented. 

6b.  Procedures  for  generating 
equivalent  multiple  forms 
are  documented. 

6c.  Pass/fail  criteria  are 

established  with  rationale 
for  setting  minimum 
standards. 

7a.  A  logical ly  derived 

hierarchical  structure  of 
the  JSEP  prerequisite 
competencies  is  produced. 

8a.  An  outline  of  unit  cost  for 
production  is  available. 

8b.  A  PERT  chart  is  available 
for  demonstrating  work  flow 
among  production  elements. 

8c.  A  production  schedule  is 
available  for  inspection. 

8d.  Quality  control  procedures 
are  in  place  for  both 
process  and  output. 


8 


9.  Develop  formative 

evaluation  procedures  to 
determine  lesson 
effectiveness  (achievement 
and  attitudes). 


10.  Design  on-going  evaluation 
procedures  to  monitor  and 
improve  JSEP  curriculum. 


9a.  The  procedures  used  to 

develop  evaluation  criteria 
and  procedures  will  be 
described. 

9b.  A  standard  evaluation 
•  instrument  is  used  for  all 
lessons  and  is  reviewed  by 
the  evaluation  committee. 

9c.  Formative  evaluation 

procedures  are  conaucted 
consistently. 

9d.'  '  Desired  outcome  standards 
are  stated. 

10a.  Evaluation  instruments  have 
been  selected  or  developed 
to  obtain  information 
pertaining  to 

(a)  achievement 

(b)  attitudes 

(c)  completion  rate 

(d)  cost 

10b.  An  evaluation  design  nes 
been  developed  by  the 
JSEP  internal  evaluation 
committee. 

10c.  Evaluation  design  has  been 
reviewed  externally  by  ARI 
and  tne  external  evalua¬ 
tion  team. 

lOd.  Data  collection  procedures 
are  designed  and  reviewed 
by  ARI,  TAGO  and  ESO's. 


OBJECTIVE  IV:  DE5I6N  AN  INSTRUCTIONAL  SUPPORT  SYSTEM  TO  FIELD  TEST 
THE  JSEP  CURRICULUM1 


Process  Steps 

1.  Suggest  an  instructional 
strategy  to  be  used  by 
personnel  who  will  administer 
remedial  instruction. 

2.  Specify  instructional 
support  personnel  required 
for  testing,  remedial 
instruction,  motivation, 
counseling,  recordkeeping, 
etc.,  within  the  limitations 
of  BSE P  II  contract 
requirements. 

3.  Specify  computer 
maintenance  support 
personnel . 


4.  Specify  library  support 
personnel  for  storage  and 
retrieval  of  instructional 
materials. 

5.  Specify  required  JSEP 
support  personnel  for 
tasks  not  directly  related 
to  delivery  of  instruction 
(e.g.,  ESO,  receptionist, 
clerk,  etc.). 


Performance  Standards 

la.  Remedial  instructional 
strategy  as  outlined  or 
suggested  for  JSEP. 


2a.  Job  titles  and  job 

descriptions  for  direct 
instructional  support 
personnel  are  specified 
for  both  the  preliminary 
and  full  scale  tryouts. 


3a.  Job  titles  and  job 

descriptions  of  computer 
maintenance  personnel  are 
recorded. 

4a.  Job  titles  and  job  descrip 
tions  cf  library  personnel 
are  recorded. 


5a.  Job  titles  and  job 

descriptions  of  related 
personnel  are  recorded. 


1.  Note:  the  Process  Steps  and  Performance  Standards  are  applicable 
to  both  the  preliminary  and  full  scale  tryouts. 


\  OBJECTIVE  V: 

\ 

^  Process  Steps 


DESIGN  A  MANAGEMENT  INFORMATION  SYSTEM  TO  DIRECT,  MONITOR 
AND  REPOR-T  STUDENT  PROGRESS  IN  JSEP 


Performance  Standards 


1.  Identify  decision  points 
for  the  JSEP  management 
system  including  decision, 
points  within  the 
curriculum. 


i 


2. 


Specify  users  of 
information  at  each 
decision  point. 


j 

3.  Assess  information 

.  requirements  for 

appropriate  personnel  at 

^C*'C,'hr' 

j  JSEP  management  system. 

4.  Develop  procedures  to  input 
and  record  information  for 
decision-making  purposes. 

i 


5.  Develop  procedures  to 
access  information  at 
decision-points. 


6.  Specify  temporary  and 
permanent  records. 


la.'  A  flow  chart  (from  the 

students  perspective)  for 
how  students  qualify  for, 
enter  in,  pass  through, 
exit  from  JSEP,  and  return 
is  presented  in  JSEP 
Management  SystenTUpera- 
tor 1 s  Manual . 

2a.  The  users  of  information 
at  each  decision  point  are 
noted  and  approved  by  ARI 
and  stated  in  JSEP  Manage¬ 
ment  System  Operator  ‘s 

ManuaTT 

3a.  The  kinds  of  information 
required  at  each  decision 
point,  as  well  as  go, 
nc-gc  criteria  a^e  stated 
in  JSEP  Management 
System  Operators  Manual. 

4a.  Procedures  for  recording 
and  storing  information 
(on-line  or  off-line)  are 
described  in  JSEP  Manage¬ 
ment  System  Operator's  Man 
ual. 

4b.  Back-up  information  system 
is  developed  to  avoid  the 
possibility  of  losing  in¬ 
formation  due  to  power 
outages,  etc. 

5a.  Procedures  for  accessing 
records  are  described  in 
JSEP  Management  System 
Operator's  ManuaTT 

6a.  Permanent  records  are 

specified,  e.g.,  student 
achievement  information, 
student  background  data 
locator  scores,  diagnostic 
scores,  summative  posttest 
scores. 
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6b. 

Temporary  records  are 
specified,  e.g.,  student  pre 
test  data,  student  practice 
data. 

Distinguish  between  open 
access  and  qualified  access 
to  information. 

7a. 

"Passwords"  are  established 
for  accessing  certain  kinds 
of  information. 

7b. 

Criteria  are  established  for 
the  issuing  of  passwords. 
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OBJECTIVE  VI:  DEVELOP  A  JOB-RELATED,  COMPUTER-BASED  PREREQUISITE 
COMPETENCY  CURRICULUM 


Process  Steps 

Performance  Standards 

1. 

Design  and  develop  a 

la. 

A  PERT  chart  is  developed 

production  system  to 

showing  how  production 

produce  computer  software, 
pretests  and  posttests,  and 

elements  are  sequenced. 

support  media. 

lb. 

Job  roles,  duties,  and 
responsibilities  are 
stated. 

lc.'  • 

A  production  management 
information  system  is 
established  with  production 
records. 

Id. 

Sufficient  hardware  is  in 
place  for  in-house 
development. 

le. 

Quality  control  review 
procedures  are  formalized. 

Identify,  secure,  and  train 

2a. 

Educational  ano  job  History 

required  production 

requirements  are  specified. 

personnel . 

2b. 

Appropriate  affirmative 
action  procedures  are 
taken. 

2c. 

Training  responsibilities 
are  stated. 

2d. 

Training  procedures  are 
documented. 

3. 

Specify  production 

3a . 

Production  schedule  is 

schedule. 

developed  and  approved  by 
ARI. 

4. 

Establish  production 

4a . 

Quality  control  criteria  are 

quality  control  criteria 
and  procedures. 

established  for  try-outs. 

4b. 

Quality  control  criteria  are 
established  for  field  test. 

5. 

Conduct  CET  tryouts. 

5a. 

At  least  3  "learners” 
participate  in  try-outs. 

5b. 

Formative  evaluation 
procedures  are  formalized. 
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Conduct  preliminary  tryout 
on  one  Army  past  (Ft.  Rucker). 


Revise  as  required. 


Develop  support  personnel 
training  materials  and 
procedures  for  field  test. 


Specify  requirements  for 
hardware  and  software 
installation  and 
maintenance  at  field  test 
site. 


5c.  Records  are  made  of  both 
learner  and  developer 
feedback. 

6a.  Conditions  of  implementation 
will  resemble  field  test. 

6b.  Observations  are  made 

regarding  achievement  and 
JSEP  system  proficiency. 

7a.  Records  are  made  of 

revisions  made  to  lessons 
and  system  specifications 
before  field  test. 

8a.  Training  procedures  for 
JSEP  support  personnel  at 
the  four  field  test  sites 
are  stated  and  approved 
by  ARI. 

8b.  Appropriate  training 

materials  are  produced  or 
collected. 

9a.  Hardware  requirements  for 
each  site  are  established 
to  take  into  account 
projected  usage  parameters 
(See  task  A  report). 

9b.  Procedures  to  store  and 

retrieve  software  are  stated. 

9c.  Contract  arrangements  and 
policies  for  hardware  ano 
software  maintenance  are 
made  among  Hazel  tine 
Corporation,  Control  Data 
Corporation,  FSU  and  ARI. 


OBJECTIVE  VII:  FIELD  TEST  A  JOB-RELATED,  COMPUTER-BASED 
PREREQUISITE  COMPETENCY  CURRICULUM 


Process  Steps 


1.  Select  two  FORSCOM  and  two 
TRADOC  bases  for  field 
test. 


2.  Interpret  Army  policy  and 
recommend  procedures  for 
determining  eligibility 
for  JSEP  field  test. 


3.  Develop  method  for 
informing  CO's  of  JSEP 
opportunity  and  its  goals. 

4.  Identify  and  secure  support 
personnel  with  necessary 
education  and  job 
experience. 

5.  T<-jin  S'jpp?'*?  povjjnnol 
to  perform  assigned  JSEP 
functions. 


6.  Establish  on-going 
evaluation  system. 


7.  Establish  a  pol icy/advisory 
committee  at  each  FORSCOM 
and  TRADOC  field  site  for 
implementing,  monitoring, 
and  evaluating  JSEP. 


Performance  Standards 

la-  Two  FORSCOM  and  two  TRADOC 
sites  are  selected. 

2a.  Criteria  for  eligibility 
for  JSEP  field  test  (e.g. , 
KOS,  locator  scores, 
diagnostic  test  scores)  are 
’  ‘  stated  and  approved  by  ARI. 

3a.  Personnel  and  media  selected 
for  informing  CO's  of  JSEP 
are  documented. 

4a.  Support. personnel  are 
identified,  hired  and 
assigned- to  JSEP  according 
to  specifications. 

5a.  The-e  '$  documentation  ox 
attendance  at  training 
sessions  held  by  CET. 

5b.  Required  competencies  are 
mastered  as  indicated  by  a 
checklist. 

6a.  Performance  indicators  are 
established. 

6b.  Policies  and  procedures  for 
evaluation  are  documented. 

6c.  Decision  makers  and 
decision  processes 
documented. 

6d.  Procedures  and  funds  for 
revisions  are  documented. 

7a.  Four  policy/advisory 

committees  are  in  place. 
Membership  is  stated  and  a 
tentative  schedule  of 
meetings  adopted. 
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s.  Articulate  exit  JSEP 
performance  standards  by 
MOS. 


J.  Recruit  and  select  soldiers 
for  JSEP  field  test. 


10.  Document  JSEP  program 
activities. 


11.  Establish  reporting 

mechanisms  for  achievement. 


12.  Install  appropriate 

hardware,  software,  and 
personnel  required  for 
field  test. 


8a.  Minimum  proficiency  levels 
on  locator  and  diagnostic 
tests  and  summative 
posttests  are  stated. 

9a.  Procedures  for  identifying 
a  pool  of  potential  "JSEP 
eligibles"  are  stated. 

9b.  Criteria  for  selecting 
"JSEP  eligibles"  are 
stated. 

9c.  Procedures  for  securing  CO 
support  to  release  "JSEP 
eligibles"  from  duty  are 
in  place. 

10a.  Amount  of  time  per  activity 
per  lesson  is  recorded  for 
learners  and  for  support 
personnel . 

11a.  Mechanisms  for  maintaining 
temporary  and  permanent 
reco-dr  2"?  *’r  p’ate. 

12a.  Hardware,  software,  and 

personnel  configuration  are 
installed  at  least  one  full 
day  prior  to  implementation 
of  field  test. 


OBJECTIVE  VIII: 


COMPARE  ALTERNATIVE  COMPUTER-BASED  DELIVERY  SYSTEMS  IN 
TERMS  -OF  EFFECTIVENESS,  EFFICIENCY,  AND  BENEFIT  TO  U.S. 
ARMY  MISSION. 


Process  Steps 

1.  Ascertain  costs  for  initial 
outlay  of  hardware. 

2.  Ascertain  costs  for 
installation. 


3.  Estimate  life  cycle 
operational  costs. 


4.  Determine  reliability  of 
each  system. 


5.  Evaluate  human  factors 

considerations  of  systems. 


6.  Determine  feasibility 
of  supplementing  JSEP 
locally. 


Performance  Standards 

la.  Initial  costs  and.  economies 
of  scale  are  reported. 

2a.  Relationship  between 
installation  costs  and 
troop  demand  are  reported. 

3a.  "  The  life  cycle  of  T ICC I T 
and  PLATO  systems  is 
estimated  and  reported. 

3b.  Replacement  costs  are 
estimated  and  reported. 

4a.  A  ratio  of  down-time/total 
time  will  be  calculated 
during  field  trial  and 
reported. 

4b.  Causes  of  oown-time  will  be 
reported. 

5a.  Attitude  scale  will  be 
selected/developed  to 
assess  human  factors  from 
learner  perspectives,  and 
instructor  and  developer 
perspectives  (e.g., 
fatigue,  eye  strain',  noise 
level,  etc.). 

5b.  Behavioral  indicators  will 
be  selected  related  to 
human  factors  (e.g.,  time 
to  learn  keyboard,  time  to 
progress  through  identical 
lessons,  break  time,  etc.). 

5c.  Results  of  attitudinal  and 
behavioral  survey  will  be 
reported. 

6a.  Compare  procedures  and 
resources  required  to 
supplement  JSEP. 
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Ascertain  costs  to  develop 

7a. 

Direct  costs  to  develop 

JSEP  software. 

JSEP  curriculum  will  be 
reported  for  T1CCIT  and  for 
PLATO. 

7b. 

Ratio  of  cost/hr.  of 
on-line  Instruction 
will  be  reported. 

Formulate  optional  hardware 

8a. 

A  report  will  detail  direct 

configuration  for  least 

costs  to  install  JSEP  at 

cost  while  holding 

multiple  levels--from  a 

effectiveness  (i.e., 

stand-alone  unit  to  a  wide 

training)  constant.  (Cost 

geographical  sha-ing  of 

Benefit  Trade-off  Analysis 

core  hardware  for  PLATO  and 

-  Task  8). 

T1CCIT. '  (See  Task  4  and 
Task  8) 

Ascertain  relationship 

9a. 

A  report  will  detail  direct 

between  JSEP  system 

costs  to  implement  JSEP  and 

elements  and  effectiveness 

show  relationship  between 

(Cost  Training 

variable  cost  elements  and 

Effectiveness  Analysis  - 
Task  9). 

achievement. 

9b. 

An  optimal  JSEP  system 
model's)  will  be 
recommenoed  for  PLATO 
and  TICCIT  in  a  final 
report. 

OBJECTIVE  IX:  EVALUATE  THE  POTENTIAL  IMPACT  OF  JSEP  ON  ARMY  JOB 
SKILLS;  EDUCATIONAL  PROGRAMS,  AND  SOLDIER  CAREER 
GOALS 


Process  Steps 

1.  Ascertain  degree  of 

development  of  JSEP  pre-. 
requisite  competencies 
attributable  to  the  JSEP 
computer  based  system. 


2.  Determine  relationship 
between  achievement  in 
JSEP  and  job  performance. 


3.  Analyze  JSEP  utility. 


Performance  Standards 

la.  Changes  in  locator  test 
scores  will  be  reported 
with  prior  education 
and  MOS  as  control 
variables. 

lb.  Change  in  summative  post- 
'  •  test  scores  of  assigned 

PC's  will  be  reported  with 
prior  education  and  MOS 
as  control  variables. 

2a.  Change  in  reading 
comprehension  of 
Soldier's  Manual  will  be 
reported  using  a  CLOZE 
test. 

2b.  A  job  performance 

measure  using  the  MOS 
task  Indicator  statements 
will  be  developed  to 
assess  changes  in  job 
performance  attributed 
to  JSEP. 

3a.  Characteristics  of  learners 
who  master  PC's  off-duty 
will  be  reported.  ■ 

3b.  Average  number  of  on-duty 
hours  to  complete  JSEP  will 
be  recorded  with  MOS  and 
site  location  as  control 
vari able. 

3c.  Usage  by  hours  of  the  day 
will  be  reported. 

3d.  Optimal  scheduling 
procedures  will  be 
recommended. 


19 


4.  Analyze  JSEP  completion 
rate. 


5.  Analyze  CO  attitudes 
toward  JSEP. 


6.  Assess  achievement  in  JSEP 
with  BSEP  II  criteria. 


7.  Assess  BSEP  II  instruction 
with  JSEP  criteria. 


8.  Analyze  attitudes  of  ESO 
and  instructional  staff. 


4a.  Completion  rate  will  be 
reported  by  MOS  with 
locator  scores,  and  prior 
education  as  control 
variables. 

5a.  CO  attitudes  will  be 
reported  regarding  con¬ 
venience,  perceived 
effectiveness,  and  value 
to  Army  mission. 

6a.  Change  in  G7  scores  will 
be  reported  with  amount  of 
JSEP  instruction,  MOS, 
and  prior  educational 
achievement  as  control 
variables. 

6b.  Change  in  TABE  scores  will 
be  reported  with  amount  of 
JSEP  instruction,  MOS, 
prior  educational  achieve¬ 
ment,  and  learning  style  as 
control  variables. 

7a.  Change  in  locator  scores 
will  be  reported  with  MOS, 
prior  educational  achieve¬ 
ment,  and  learning  style  as 
control  variables. 

7b.  Changes  in  summative 
posttest  scores  will  be 
reported  with  prior 
educational  achievement 
and  MOS  as  control 
variables. 

7c.  Changes  in  job  perfor¬ 
mance  using  JPM  with 
task  indicator  state¬ 
ments  will  be  used  to 
assess  changes  in  job 
performance  attributed 
to  BSEP  II. 

8a.  ESO  and  instructional 
staff  attitudes  toward 
JSEP  wil 1  be  reported 
regarding  convenience, 
effectiveness  and  value  to 
Army  mission. 
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9.  Analyze  learner  attitudes.  9a.  Attitudes  of  troops  toward 

OSEP  in  terms  of  perceived 
effectiveness,  usefulness, 
and  enjoyment  of  learning 
in  environment  will  be 
reported. 
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OBJECTIVE  X:  EXPLORE  THE  FEASIBILITY  OF  USING  JSEP  TO  AWARD  HIGH 
SCHOOL  CREDIT  OR  CREDENTIALS 


Process  Steps  Performance  Standards 


Analyze  relationship  be¬ 

la. 

Examine  RCA  validity 

tween  performance  on  the 
Locator  Test  and  on  the 

research. 

GED. 

lb. 

Validate  RCA  research  on 
JSEP  field  test  population 
by  reporting  correlations 
between  locator  and  GED 
for  JSEP  population. 

2a.  Results  of  a  content 

analysis  will  be  reported 
using  SME's  to  rate 
similarity  between  JSEP 
PC's  ano  GED  competencies. 

3a.  Taking  a  sample  of  high 
school  courses  in  Florida, 
report  degree  of  overlap 
between  PC's  and  course 
content. 

3b.  Taking  a  sample  of 

accredited  high  school 
correspondence  courses, 
report  degree  of  overlap 
between  PC's  anc  course 
content. 

4a.  Policy  statement  is 

developed  with  supportive 
research  documentation. 


5a.  Policy  statements  will  be 
developed  with  supporting 
research  documentation. 


4.  Prepare  policy  recom¬ 
mendation  regarding 
performance  on  Locator 
and  awarding  of  GED. 

5.  Prepare  policy  recom¬ 
mendations  regarding  JSEP 
PC  achievement  on 
Diagnostic  Tests  and  the 
awarding  of  high  school 
courses  and  credits. 


2.  Compare  JSEP  prerequisite 
competencies  with  GED 
competencies. 


3.  Compare  JSEP  prerequisite 
competencies  with  high 
school  courses  and  com¬ 
petencies. 
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Present  policy  recom¬ 

6a. 

Target  audiences  are 

mendations  to  regional  and 
national  accrediting 

identified. 

societies. 

6b. 

Briefings  with  members  of 
accrediting  societies  are 
arranged. 

6c.  Outcomes  of  briefings  are 
reported. 


OBJECTIVE  XI:  PREPARE  SPECIFICATIONS  FOR  THE  DISSEMINATION  OF  JSEP 
ARMY-WIDE 


Process  Step 

1.  Develop  a  JSEP  system 
model (s)  based  on  cost 
and  effectiveness  data 
from  field  test. 

2.  Specify  PC's  for  which  JSEP 
has  demonstrated 
effectiveness. 

3.  Recommend  a  needs  assess¬ 
ment  procedure  to  identify 
PC  deficiencies. 

4.  Develop  procedure  to 
estimate  costs  to  remove 
PC  deficiencies  through 
use  of  JSEP. 


Performance  Standards 

la.  JSEP  system  model (s)  are 
developed  which  optimize 
costs  and  effectiveness. 


2a.  PC  with  demonstrated 
effect  sizes  beyond 
chance  are  identified. 

3a.  An  efficient  needs 

assessment  procedure  will 
be  recommended  in  a  report 

4a.  A  predictive  cost  model 
will  be  reported  to 
estimate  cost  to  remove  PC 
deficiencies. 


APPENDIX 


LIST  OF  ACRONYMS 

ASVAB  -  Army  Services  Vocational  Aptitude  Battery 

ARI  -  Army  Research  Institute 

BSEP  -  Basic  Skills  Education  Program 

CBI  -  Computer  Based  Instruction 

CET  -  Center  for  Educational  Technology,  Florida  State  University 

CLOZE  -  Test  for  reading  level 

CTEA  -  Cost  and  Training  Effectiveness  Analysis 

ESO  -  Education  Services  Officer 

ETS  -  Educational  Testing  Service,  Princeton,  N.J. 

FORSCOM  -  United  States  Army  Forces  Command 

FSU  -  Florida  State  University 

6ED  -  General  Educational  Development  Test 

GT  -  General  Technical  scores  from  the  ASVAB 

JSEP  -  Job  Skills  Education  Program 

JPM  -  Job  performance  measure 

MOS  -  Military  Occuoation  Soecialtv 

PC  -  Prerequisite  Competency 

RCA  -  Radio  Corporation  of  America 

SME  -  Subject  matter  experts 

SSP  -  Scientific  Service  Project 

TABE  -  Test  Df  Adult  Basic  Education 

TAGO  -  The  Adjutant  General's  Office 

TICCIT  -  Hazeltine  Computer  System 

TRADOC  -  United  States  Army  Training  and  Doctrine  Command 


